switch titiler.xarray to obstore+zarr and add default application #1253

vincentsarago · 2025-10-29T12:29:08Z

This PR does:

create a new Opener function using Obstore and Zarr (only for ZARR Dataset)
switch the default reader to open_zarr opener
add Zarr-python and Obstore as dependencies
create a default application (for Zarr)
remove python 3.10 support

ToDo

add tests for the default application
update documentation
add example for application using fsspec opener

vincentsarago · 2025-10-29T12:30:39Z

src/titiler/xarray/tests/test_io_tools.py

    """test reader."""
    src_path = protocol + os.path.join(protocol, prefix, filename)
-    with Reader(src_path, variable="dataset") as src:
+    with Reader(src_path, variable="dataset", opener=fs_open_dataset) as src:


Reader(src_path,, opener=fs_open_dataset, ...) is the same as titiler.xarray.io.FsReader

src/titiler/xarray/tests/test_io_tools.py

vincentsarago · 2025-10-29T12:32:33Z

src/titiler/xarray/titiler/xarray/io.py

+
+    # Fallback to Zarr
+    else:
+        store = zarr.storage.FsspecStore.from_url(


removed support for zarr-python 2.0

vincentsarago · 2025-10-29T12:47:27Z

.github/workflows/ci.yml

    strategy:
      matrix:
-        python-version: ['3.10', '3.11', '3.12', '3.13']
+        python-version: ['3.11', '3.12', '3.13']


because titiler.xarray cannot support 3.10, it's just easier to also switch all package to >=3.11

hrodmn

Thanks for adding titiler.xarray.main! I ran uv run uvicorn titiler.xarray.main:app and it worked great out of the box, but I did have to set AWS_SKIP_SIGNATURE=True to get anonymous reads by default. I wonder how we might configure that type of setting at the application level.

src/titiler/xarray/titiler/xarray/io.py

hrodmn · 2025-10-29T14:19:18Z

src/titiler/xarray/titiler/xarray/io.py

+        if "region" not in kwargs and infer_region:
+            # infer region or fallback to env variables
+            region_name_env = (
+                os.environ.get("AWS_REGION", os.environ.get("AWS_DEFAULT_REGION"))
+                or None
            )
-        else:
-            store = fsspec.filesystem(protocol).get_mapper(src_path)
+            config["region"] = _find_bucket_region(parsed.netloc) or region_name_env


maybe throw an http exception if region can't be discovered

hrodmn · 2025-10-29T14:22:52Z

src/titiler/xarray/titiler/xarray/io.py

+        fs = fsspec.filesystem(protocol, **kwargs)
+        ds = xarray.open_dataset(fs.open(src_path), **xr_open_args)
+
+    # Fallback to Zarr


would it make sense to take zarr support out of this opener and have the application layer choose xarray_open_dataset or open_zarr? maybe it is nice to make this one reader that can open anything though.

in theory you could but I think for compatibility issue it's better to keep everything separated.

maybe it is nice to make this one reader that can open anything though.

User could do it if they need

hrodmn · 2025-10-29T14:23:15Z

src/titiler/xarray/titiler/xarray/main.py

vincentsarago · 2025-10-30T11:27:54Z

Thanks for adding titiler.xarray.main! I ran uv run uvicorn titiler.xarray.main:app and it worked great out of the box, but I did have to set AWS_SKIP_SIGNATURE=True to get anonymous reads by default. I wonder how we might configure that type of setting at the application level.

That's a main issue with Obstore, it doesn't do any guesses for the AWS Region or if the data is public (ref: developmentseed/obstore#321)

We're already have a hack to guess the region, maybe we could implement something to guess if we need to set skip_signature 🤷

hrodmn · 2025-10-30T11:46:33Z

We're already have a hack to guess the region, maybe we could implement something to guess if we need to set skip_signature 🤷

It isn't pretty, but could we add anonymous and region as optional args in the tile endpoints, ~~then add an endpoint like /storage that would do the guess and check to find out those parameters? It doesn't seem right to have to do the same guess and check for each tile request.~~

Actually, if we add anonymous and region as query parameters and return informative 400 errors to the user, we could prompt them to set anonymous=True and region={region} depending on the error we get back from obstore.

vincentsarago · 2025-10-30T13:10:14Z

The open_zarr has a LRU cache so ideally it shouldn't be called too many time for a specific Zarr dataset

we could add region as query parameter but for the anonymous it's a bit different. If the environment has correct AWS credentials it will work fine on public/private dataset, the skip_signature=True is only useful if there are no credentials in the env

vincentsarago · 2025-10-30T13:58:53Z

Ok I've changed some logic

As mentioned before s3:// url will be considered private so S3 credentials should be present in the environment.

if user want to access public s3 object they would need to use https:// url (we will set skip_signature=True)

vincentsarago · 2025-10-30T14:00:51Z

src/titiler/xarray/titiler/xarray/io.py

+                    if not expr.groupdict().get("region"):
+                        config["region"] = (
+                            _find_bucket_region(bucket) or region_name_env
+                        )


This is to support old style URL https://{bucket}s3.amazonaws.com/...

maxrjones · 2025-11-03T14:54:05Z

FYI this type of wrapper - virtual-zarr/obspec-utils#1 - provides a way to use xarray + obstore with libraries that expect file handles (e.g., netcdf4/h5netcdf). That would help if the goal is just to limit dependencies, but wouldn't help if the goal is to simplify code complexity by removing support for data formats other than zarr.

switch titiler.xarray to obstore+zarr and add default application

2821041

vincentsarago commented Oct 29, 2025

View reviewed changes

src/titiler/xarray/tests/test_io_tools.py Outdated Show resolved Hide resolved

vincentsarago commented Oct 29, 2025

View reviewed changes

remove python 3.10 and recreate uv.lock

1b343c9

vincentsarago commented Oct 29, 2025

View reviewed changes

vincentsarago requested review from hrodmn and maxrjones and removed request for hrodmn October 29, 2025 13:03

hrodmn reviewed Oct 29, 2025

View reviewed changes

kylebarron mentioned this pull request Oct 29, 2025

S3Store.from_url fails with https URL developmentseed/obstore#589

Open

vincentsarago added 3 commits October 30, 2025 12:17

better handle aws s3 http urls

ecef2f9

update from main

98706f5

update from comment

9277098

lower bounds for s3fs

603bc5a

assume https s3 object are public

5d12df2

vincentsarago commented Oct 30, 2025

View reviewed changes

vincentsarago requested a review from hrodmn October 30, 2025 14:17

switch titiler.xarray to obstore+zarr and add default application #1253

Are you sure you want to change the base?

switch titiler.xarray to obstore+zarr and add default application #1253

Uh oh!

Conversation

vincentsarago commented Oct 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

ToDo

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hrodmn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vincentsarago commented Oct 30, 2025

Uh oh!

hrodmn commented Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vincentsarago commented Oct 30, 2025

Uh oh!

vincentsarago commented Oct 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxrjones commented Nov 3, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

vincentsarago commented Oct 29, 2025 •

edited

Loading

hrodmn commented Oct 30, 2025 •

edited

Loading